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private school voucher component of the Florida A+ Accountability Program was unconstitutional, 
eliminating the threat of having these students and funds leaving to attend private schools. This 
exogenous shock allows us to test whether private voucher threats and the funding tied to these 
students led to increases in student achievement. We find no evidence that the private school 
voucher threats drive academic improvement beyond what is seen in schools when this private 
school voucher threat is removed. 
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Estigma y sin sanciones: La (falta de) impacto de los vales escolares ( voucher) para escuelas 
privadas en el rendimiento estudiandl. 

Resumen: En el marco del Programa de Evaluation Florida A +, las escuelas de la Florida se 
clasiflcan en funcion del rendimiento de los estudiantes en el Examen de Evaluation Integral de la 
Florida. Anteriormente, cuando las escuelas recibian una calificacion negativa en un lapso de cuatro 
anos, a los estudiantes de esas escuelas se les ofrecian vales escolares ( voucherj que los padres y 
tutores podrian utilizar para transferir a los estudiantes a una escuela publica traditional u otra 
privada. En enero de 2006 la Corte Suprema de la Florida declaro que el voucher para las escuelas 
privadas del Programa de Responsabilidad Florida A + era inconstitucional, lo que elimina la 
amenaza de que esos estudiantes y los fondos fueran transferidos las escuelas privadas. Este choque 
exogeno nos permite comprobar si las amenazas de los vouchers y la financiacion vinculada a estos 
estudiantes llevaron a aumentar el rendimiento estudiantil. No encontramos ninguna evidencia de 
que las amenazas de los vouchers generara mejorias academicas mas alia de lo que se observo en las 
escuelas cuando se elimino la amenaza de los vouchers. 

Palabras clave: vales educativos {vouchers)-, rendition de cuentas; politica educativa; 
investigation cuantitativa 

Estigma sem san§oes: A (falta de) impacto dos vales de escolares ( voucher ) para 
escolas particulares no desempenho dos alunos. 

Resumo: No ambito do programa de avalia^ao Florida A +, as escolas da Florida sao 
classificados com base no desempenho do estudante na Avalia^ao Global da Florida. 
Anteriormente, quando as escolas receberam uma avalia^ao negativa em um periodo de 
quatro anos, aos alunos dessas escolas foram oferecidos vales escolares {voucher) que os pais 
e responsaveis poderiam utilizar para transferir os alunos para uma escola publica 
tradicional ou outra privada. Em janeiro de 2006, o Supremo Tribunal da Florida decidiu 
que o programa de vales {voucher) para escolas particulares. do programa Florida A + era 
inconstitucional, eliminando a amea^a de que esses alunos e os fundos fossem transferidos 
para escolas particulares. Este choque exogeno nos permite verificar se a amea^a dos 
vouchers e financiamentos vinculados a esses alunos levaram a aumentar o desempenho 
academico dos alunos. Nos nao encontramos nenhuma evidencia de que a amea^a de 
vouchers gerara melhorias academicas para alem do que foi observado nas escolas onde a 
amea^a dos vouchers foi removida. 

Palavras-chave: vales escolares (vouchers); presta^ao de contas; politica educacional; 
pesquisa quantitativa. 


Introduction 

School systems have addressed lagging student achievement with a variety of education 
reforms and accountability policies. The No Child Left Behind Act of 2001 (NCLB) has made high- 
stakes, accountability-based standardized testing paired with school choice policies increasingly 
common throughout the United States (Betebenner, Flowe, & Foster, 2005). Though not exactly 
identical, many states have implemented accountability-based programs quite similar to Florida’s A- 
Plus Accountability Program (A+) since the enactment of NCLB (Camilli & Bulkley, 2001). Since 
the A+ program precedes NCLB and was the model applied to the entire nation with the passage of 
NCLB, the vast panel data available from Florida has allowed researchers to explore the 
consequences of NCLB-type reform programs. 
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Under A+, schools are assigned letter grades (A — F) and receive sanctions or rewards based 
on student performance on the Florida Comprehensive Assessment Test (FCAT). Schools’ grades 
are determined by an algorithm that includes their students’ proficiency levels and test score gains. 1 
When earning a second failing grade (F) within four years, the state places a school under state 
sanctions, provides additional resources, and offers vouchers to students assigned to attend these 
schools. 

From the perspective of market-based education reformers, private school choice policies 
are fundamentally necessary for improving public school performance through competition (e.g., 
Friedman, 1962). Chubb and Moe (1990) contend that failing institutions are not likely capable of 
effectively implementing reform from within when it was likely the actions of those in charge that 
brought about the failing status (or at least the inability to make improvements). Chubb and Moe 
(1990) even make a case for “the notion that choice is a panacea” (p. 217). However, despite this 
assertion, there is also a basis for choice alone being insufficient if parents do not have information 
regarding school performance and that uniting choice and test-based accountability is necessary for 
improving outcomes (Hastings & Weinstein, 2008). Therefore, state policymakers might more 
successfully improve school performance by motivating district and school administrators with 
sanctions and incentives in addition to choice (Bettebenner, Howe, & Foster, 2005). 

While traditional public schools are typically already competing with charter and private 
schools for students, private school vouchers can expand schooling options, especially to students 
from low socioeconomic status (SES) households. Moreover, the private school vouchers also 
provide a threat to school districts’ funds that are tied to student enrollments. Greene and Winters 
(2003) elaborate on this concept: “The theory behind the A+ Program is that chronically failing 
public schools will have an incentive to improve if they must compete with other schools for 
students and the funding they generate” (p. 1). However, Carnoy (2001) contends that the need for 
choice policies within test-based accountability programs is likely overstated, and Ladd and Glennie 
(2001), directly addressing Greene’s (2001) earlier study on this issue, find that “Greene has 
inappropriately attributed the differential gains to the voucher program rather than to the other 
effects of being labeled a failing school, such as shame, increased scmtiny, and possibly additional 
resources” (p. 49). 

As state education policymakers continue to assess ways to increase standardized test score 
performance at lower-performing schools, it is critical to examine the effectiveness of particular 
components within these accountability programs. Examining one particular component of an 
accountability program is often difficult as multiple policies are typically implemented 
simultaneously as part of a comprehensive reform plan. In instances where these types of private 
voucher sanctions lead to positive results, it could be that NCLB-type accountability programs could 
have produced comparable effects without the additional threat of a private school voucher sanction 
and loss of funds (Harris, 2001). For example, in addition to the provision of private school 
vouchers, states also more closely supervise and allocate additional resources to sanctioned schools. 
These schools also receive a great deal of public scrutiny from parents and the media as a result of 
‘ ‘underperforming’ ’. 

To better estimate the effect of a private school voucher sanction within the context of a 
state accountability system, we take advantage of a major change to Florida A+ that provides a 
natural experiment that allows us to isolate this sanction’s impact. In January of 2006 the Florida 
Supreme Court declared the private school voucher component of A+ was unconstitutional. 


1 A comprehensive overview of Florida’s school grading system over the years is available at the Florida 
Department of Education’s (2013) “School Grades: School Accountability Report Links.” 
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immediately eliminating the private school voucher threat 2 3 for students attending schools designated 
as failing (Bush v. Holmes, 2006). 1 This immediate, exogenous change allows us to test the 
effectiveness of the private school voucher threat under a state’s test-based accountability program, 
independent of the other accountability policies. Therefore, the purpose of this study is to examine 
whether the threat of private school voucher sanctions produces significant effects for students 
attending “lower-performing” public schools. 

Testing the effectiveness of this private school voucher threat provides insight into whether 
the threat of losing students and funding incentivizes public schools to bring about increases in 
student achievement on state standardized tests. For this study it is important to note that we rely on 
the assumptions that producing gains on these tests is a valued objective of the designers of this 
accountability program and that standardized test score improvements represent a meaningful 
educational outcome. It is important to note that prior research has demonstrated that there are 
concerns with interpreting high-stakes assessment results, especially when these results are partnered 
with accountability measures (e.g., Jones, 2007; Nichols & Berliner, 2007; Rubin, 2011; Wang, 
Beckett, & Brown, 2006). 4 Therefore, while our study is specifically intended to inform the 
discussion of whether a particular aspect of state-based accountability systems are successful in 
driving changes in standardized test scores, these findings do not speak to or further examine the 
issue of whether these outcomes necessarily signify real or desirable benefits in terms student 
learning. 

We find no evidence to support the claim that the threats associated with a private school 
voucher program produces test score gains beyond what is seen in lower-performing schools 
without this sanction in place. This finding has implications with regard to the design and 
implementation of state school accountability policies and programs. Private school vouchers are 
not necessarily harmful to the academic growth of public school students who are attending schools 
most likely to feel the pressures of such a sanction. However, this sanction, at least within the 
context of an NCLB-inspired accountability program, does not seem to significantly increase test 
score results beyond the improvements that occur without a private school choice sanction in place. 

In the next section we provide an overview of the previous literature on test-based 
accountability systems as well as private school voucher sanctions. Next, we explain the methods 
and data used to conduct this study. Then, we present our findings and discuss their implications. 
Finally, we conclude with a summarization of some key takeaways in addition to some limitations 
and suggestions for further analyses on this subject. 


2 When referring to the “threat” of a sanction, we specifically mean schools that have received one F within 
the past four years and would have vouchers offered to their students should they earn another F. 

3 The public school voucher option still exists, but the reality of public school organization in Florida likely 
makes it ineffective in terms of the possibility of a district losing the funding tied to its students. If students 
attend another public school in their home district, there are no financial implications for the district. Each 
school district in Florida encompasses an entire county, making it difficult for students to attend a school in 
another district. For the 2012-13 school year only 32 elementary and middle schools in 14 out of 67 districts 
were participating in Florida’s Opportunity Scholarship public school choice option (Florida Department of 
Education, 2012). 

4 For an overview of the primary arguments in accountability-based reforms, we refer readers to Wang, 
Beckett, and Brown (2006). 
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Prior Research 

Standardized Test-Based Accountability 

Prior research has shown that NCLB-inspired, test-based accountability programs have 
generally produced significant increases in student achievement as measured by test score gains 
(Braun, 2004; Carnoy & Loeb, 2002; Dee & Jacob, 2011; Hanushek & Raymond, 2005; Lee, 2008; 
Rosenshine, 2003). When examining the impact of what are typically the most severe sanctions for 
underperforming schools, Figlio and Rouse (2006) and Greene (2001) find that accountability 
programs tend to be effective at increasing student performance for those students attending lower- 
performing schools. Subsequent analyses call the magnitude and interpretations of these gains or the 
mechanisms by which schools achieve these gains into question (e.g., Amrein & Berliner, 2002; 
Amrein-Beardsley & Berliner, 2003; Camilli & Bulkley, 2001; Koretz & Barron, 1998; Kupermintz, 
2001; Nichols, Glass, & Berliner, 2006). However, some studies provide empirical evidence that 
seems to suggest that these programs produce lasting and potentially meaningful improvements in 
student achievement on state-based assessments (e.g., Chiang, 2009; Rouse, Hannaway, Goldhaber, 
& Figlio, 2007; West & Peterson, 2006). 

A common concern with test-based accountability programs is that school administrators 
and faculties respond to incentives in ways that only artificially improve student achievement as 
measured on high-stakes assessments (e.g., Chakrabarti, 2007; Jacob & Levitt, 2003; Kupermintz, 
2001). Some of these responses include schools putting more emphasis on producing gains with 
students near proficiency cuts (Neal & Schanzenbach, 2010) and disproportionately excluding low- 
achieving students from testing (Heilig & Darling-Hammond, 2008). There is also evidence that 
suggests school accountability programs have substantially changed the way teachers view 
assessments in general (Brown & Harris, 2009). Where assessment was originally a way to decide 
how best to improve classroom instruction, it is now seen primarily as a school accountability tool. 
The results of these studies can help explain initial school-level responses as well as the influences 
that these stigmas or threats may have over school behaviors. 

There is some evidence that sanctioned schools in test-based accountability programs do in 
fact change their practices in meaningful ways such as allocating more resources to assist the 
instmction of lower-performing students (e.g., Booher-Jennings, 2005; Rouse, Hannaway, 
Goldhaber, & Figlio, 2013). However, other studies show that these score gains on states’ high- 
stakes tests may not necessarily signify true learning gains in terms of other assessments used to 
measure student success (e.g., ACT, SAT, and NAEP) (e.g., Amrein & Berliner, 2002; Figlio & 

Loeb, 2011; Klein, Hamilton, McCaffrey, & Stecher, 2000). The lack of translation is plausibly the 
result of changes in stakes that cause schools and teachers to produce artificial gains (e.g., 
Chakrabarti, 2007; Jacob & Levitt, 2003; Neal & Schanzenbach, 2010). Therefore, while schools 
appear to substantially respond to test-based accountability programs and even produce lasting gains 
in student achievement, the extent to which these responses actually benefit student learning is 
strongly contested. 

Private School Voucher Sanctions 

Several studies have found that the voucher sanctions and corresponding financial incentives 
are key contributors to the improvements that occur when accountability programs introduce 
competition to underperforming schools (e.g., Greene, 2001; Greene & Winters, 2003; West & 
Peterson, 2006). These studies appear to align with a general finding that competition from vouchers 
and tax credit scholarship programs tend to lead to system-wide improvements in student 
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achievement (Egalite, 2013). Specifically, Greene and Winters (2003) conclude that schools 
experiencing greater pressures (i.e. schools already under private school voucher sanctioning or very 
close to sanctioning) produce greater gains relative to similar schools not feeling the same pressures. 
They attribute these differences in achievement gains to the magnitude of a private school voucher 
threat and suggest it as evidence for these threats independently driving gains in a test-based 
accountability system. West and Peterson (2006) find that public school choice stemming from 
failure to meet Adequate Yearly Progress (AYP) under NCLB, was not producing results 
comparable to those of the private school vouchers issued under Florida’s A+ Program. This 
finding could suggest that private schools vouchers play a significant, independent role in increasing 
student achievement levels in lower-performing schools. This role can likely be attributed to the fact 
that private school vouchers put more pressure on district funds than public school choice. 

Other studies, however, have attributed the positive results from accountability programs to 
schools’ responses to other aspects of these types of sanctions (e.g., Figlio & Rouse, 2006; 
Goldhaber & Flannaway, 2004; Ladd & Glennie, 2001; Mintrop & Trujillo, 2005). For example, 
Figlio and Rouse (2006) find that the addition of the voucher sanction does not significantly 
improve student performance. They conclude that other sanctions, namely the state-provided 
additional resources and support as well as the stigmatizing of poor-performing schools, have 
greater influence over improvements in student achievement. Goldhaber and Flannaway (2004) 
conduct black box-type analyses to get a better sense of school-level responses to test-based 
accountability sanctions. These types of analyses are especially informative when it comes to 
examining intermediate outcomes that potentially have substantial roles on policy outcomes. They 
survey teachers and administrators to try and get a better sense of how school faculty, staff, and 
administrators were reacting to A+ incentives. Goldhaber and Flannaway (2004) conclude that 
school personnel are significantly more conscious of the reputation that accompanies their schools' 
grades as opposed to the possibility of students acquiring private school vouchers. Since teachers 
and administrators appear to be more aware of public scrutiny than private school choice sanctions, 
there is reason to believe that the former does more to directly motivate school personnel than the 
latter. This finding potentially informs the results of our study and is therefore further explored in 
the greater test-based accountability context in the discussion of our results. 

We hope to further inform this discussion by examining how the sudden discontinuation of 
a private school voucher sanction affects test score performance for students attending schools that 
have been deemed to be underperforming. Unlike many prior studies, we are able to examine how 
the same population of students performs both with and without the presence of this sanction. 
Moreover, the fact that we examine the discontinuation rather than the introduction of this policy 
allows us to analyze whether the lack of impacts from these sanctions is attributable to results that 
only materialize after some time. In the following sections, we describe our data and methods, 
provide the results of our analyses, discuss the implications of our findings as well as limitations, and 
conclude with takeaways for state policymakers as well as proposals for future research. 

Empirical Framework 


Data 

This study utilizes a dataset provided by the Florida Department of Education containing 
test scores in reading and math as well as demographic information for all students enrolled in 
Florida public schools for grades 3-10 from 2002 - 2008. The student level data are supplemented 
with school level information on the point total and grade earned by each school over the same time 
period. This dataset consists of over 8 million student/school/year observations. For this analysis 
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we focus on a small subset since we are utilizing a time differences in regression discontinuity (RD) 
model. 3 We use only students in grades 4-6 attending schools that earned a grade of D or F within 
0.5 standard deviations of the D/F break point. The vast majority of students attending these 
schools were eligible for free and reduced-price lunch (90 percent) and were either Black or Hispanic 
(72 percent), and 21 percent were identified as students with special needs. The full set of descriptive 
statistics for our analyses are broken out by grade level and are presented in Appendices A through 
C. 

Methodology 

Our methodology for analyzing the effect of the private voucher sanctions (or threats) on 
student math and reading achievement is similar to the Rouse et al. (2013) strategy. Since the point 
system used to grade schools changed the same year the voucher threat was removed (Florida 
Department of Education, 2010a), we are unable to use a cubic function of the schools’ point totals 
to account for school quality. To measure only the impact of a school receiving an F and minimize 
picking up regression to the mean tendencies, we employ an RD approach and limit the sample to 
students in schools within 0.5 standard deviations of the D/F point breakpoint. For the 2002 - 2005 
school years in our dataset, this includes D-graded schools earning between 280 and 305 points, and 
schools earning an F between 254 and 279 points. The school grading scale changed for the 2006 
school year, so for later years D-graded schools are included if they earn 395 - 428 points and F- 
graded schools are included when earning 362 - 394 points. These restrictions leave us with several 
thousand student-level observations from which we estimate the following model: 

T ist = o + i/(T«m) + 2 X* + 3 Y > + + 5 F*Threat + sr 

Where T indicates the test score for student i in school s during year f]J[ T ist4 ) is a cubic function of a 
student’s test score in the prior school year; X lt is a vector of observed student characteristics in year 
/, Y is a year indicator (the calendar year of the fall semester); F st indicates the school attended by 
student i in year / received an F in the previous school year; F*Threat indicates the school earned an 
F in the previous year and the private school voucher threat was still present; and st is a stochastic 
error term clustered by school and year. 

We estimate this model for each level, from fourth to sixth grade, for reading and math test 
scores separately. Third grade is excluded because it is the first tested grade level, so no previous 
year test scores are available for prediction. Students above grade six are excluded since there were 
very few junior high and high schools earning F grades under A+. Any impact of an F we find for 
secondary schools is likely to be school specific rather than a result of the statewide policy. We 
estimate a separate equation for each grade to allow for different achievement growth patterns by 
grade level as the curriculum, test difficulty, and other cohort level conditions vary. The coefficients 
of interest are 4 which estimates the overall effect of receiving a high F as opposed to a low D, and 
5 , which estimates the differential effect of receiving an F when voucher threats are still present. 

Results 

The results from estimating the model using math test scores are presented in Table 1. As 
expected, previous test scores and student demographic controls have a highly statistically significant 


5 For an empirical description of regression discontinuity design see Wooldridge (2002). 
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influence on current test scores for every grade level. 6 7 We estimate two sets of models, one set (1-3) 
with 2008 included and another set (4-6) with 2008 excluded since the point accumulating school 
grading system was altered that year.' In all models the excluded year is the 2005-2006 school year. 
This is the year during which the Florida Supreme Court ruling (Bush v. Holmes, 2006) was 
announced and it is not clear whether schools anticipated a change to the private voucher threat 
while the lawsuit was pending, and the point accumulating system was also altered the same year. 

For math scores, the coefficient on the school receiving an F in the previous year is positive 
and statistically significant in four of the six equations, indicating that students in schools recently 
earning a high F experience larger test score gains than students in schools recently earning a low D. 
The coefficient on the interaction term for a school with an F grade while the threat was in place is 
negative but statistically insignificant, indicating the threat of students receiving private school 
vouchers has no additional impact beyond the other consequences that come with receiving an F. 

Table 1 


Stigma and Voucher Sanction Threat Impacts on Math Scores 


Variables 

(1) 

Grade 4 

(2) 

Grade 5 

(3) 

Grade 6 

(4) 

Grade 4 

(5) 

Grade 5 

(6) 

Grade 6 

F s ,.i 

25.82*** 

22.21** 

11.16 

20.37** 

18.53* 

9.126 


[8.798] 

[9.060] 

[9.449] 

[8.495] 

[10.71] 

[9.289] 

F*Threat 

-14.29 

-28.48 

-8.855 

-8.438 

-23.92 

-7.129 


[21.60] 

[23.64] 

[19.24] 

[21.91] 

[24.14] 

[19.19] 

(T isl _i) 

lag_Math 

-1.100*** 

-1.861*** 

-1.872*** 

-0.972*** 

-1 731*** 

-1.895*** 

lag_Math2 

[0.112] 

0.001*** 

[0.215] 

0.002*** 

[0.178] 

0.002*** 

[0.136] 

0.001*** 

[0.225] 

0.002*** 

[0.191] 

0.002*** 

lag_Math3 

[0.000] 

-0.000*** 

[0.000] 

-0.000*** 

[0.000] 

-0.000*** 

[0.000] 

-0.000*** 

[0.000] 

-0.000*** 

[0.000] 

-0.000*** 

~\T 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

Xit 

FRL 

-18 99*** 

-10.28* 

-24 93*** 

-21.68*** 

-7.803 

-27.56*** 

Disability 

[5.414] 

-55.52*** 

[5.610] 

-68.47*** 

[5.041] 

-56.84*** 

[5.987] 

-56.06*** 

[6.430] 

-63.62*** 

[5.036] 

-53.33*** 

Male 

[4.726] 

-2.220 

[6.104] 

-6.797** 

[5.052] 

-12.48*** 

[5.308] 

-0.537 

[6.522] 

-11 43*** 

[4.734] 

-12.23*** 

Minority 

[3.004] 

-37.10*** 

[3.023] 

-30.53*** 

[2.712] 

-5.188 

[3.549] 

-43.31*** 

[3.423] 

-35.93*** 

[2.976] 

-3.567 


[6.982] 

[5.114] 

[6.797] 

[7.637] 

[5.861] 

[7.065] 


6 One interesting result is that student’s race does not significantly predict math test scores in the grade six 
equations. 

7 Florida implemented a change in their school grading procedure that included science test scores and an 
emphasis on the gains of lowest performing math students were now part of the grading system. Therefore, 
we include additional specifications as a robustness check to determine whether including this year influences 
results. 
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Table 1 (cont’d.) 


Stigma and Voucher Sanction Threat Impacts on M 

lath Scores 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

Variables 

Grade 4 

Grade 5 

Grade 6 

Grade 4 

Grade 5 

Grade 6 

Y t 

Y_2003 

17.51 

-0.306 

-51 77*** 

16.55 

-0.684 

-52.37*** 


[13.54] 

[14.39] 

[17.51] 

[13.49] 

[14.41] 

[17.48] 

Y_2004 

4.000 

-5.385 

-13.85 

2.590 

-6.135 

-14.17 


[13.87] 

[13.09] 

[14.75] 

[13.80] 

[13.03] 

[14.68] 

Y_2006 

15.12 

5.911 

-22.50 

15.70 

6.216 

-22.77 


[14.74] 

[13.50] 

[16.87] 

[14.82] 

[13.46] 

[16.99] 

Y_2007 

21.23** 

-31.34*** 

-21.17* 

22.00** 

-31.31*** 

-20.90* 


[8.818] 

[11.50] 

[11.72] 

[8.674] 

[11.56] 

[11.68] 

Y_2008 

35.22*** 

-11.14 

22.55 





[11.78] 

[11.02] 

[17.43] 




Constant 

1,348*** 

1,720*** 

1,817*** 

1,312*** 

1,660*** 

1,833*** 


[47.46] 

[93.37] 

[73.60] 

[54.50] 

[98.86] 

[77.21] 

Observations 

13,281 

13,493 

19,796 

10,360 

10,596 

18,038 

R-squared 

0.543 

0.554 

0.585 

0.525 

0.544 

0.580 


Note: Robust standard errors in brackets. *** - significant at p < 0.01, ** -p < 0.05, * -p < 0.10 


The results obtained from the data on reading test scores are found in Table 2. As with the 
math results, previous test scores and demographic controls are statistically significant and in the 
direction consistent with other results. Again, we see positive results when schools earned an F in 
the previous year; although, they are only statistically significant in three of the six equations. When 
the F grade is interacted with the existence of the voucher threat, the coefficients are negative, have 
an absolute value greater than the F coefficient, and are statistically significant in four of the six 
equations. This result suggests that the threat of private school voucher sanctions actually 
contributes to lower student test performance in reading. 


Table 2 


Stigma and Voucher Sanction Threat Impacts on \ 

■Leading Scores 





(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

Variables 

Grade 4 

Grade 5 

Grade 6 

Grade 4 

Grade 5 

Grade 6 

F s ,-i 

22.09*** 

12.27* 

4.559 

18.13* 

11.18 

4.396 


[8.295] 

[7.103] 

[10.70] 

[9.412] 

[8.197] 

[10.79] 

F*Threat 

-30.36** 

-35.52* 

-20.40 

-26.19* 

-34.35* 

-20.35 

(Ti s ,_i) 

[14.59] 

[20.73] 

[13.81] 

[15.19] 

[20.32] 

[13.82] 

lag_Reading 

-0.308*** 

-0.989*** 

-1.480*** 

-0.329*** 

-0.900*** 

-1.502*** 


[0.084] 

[0.100] 

[0.097] 

[0.096] 

[0.117] 

[0.111] 

lag_Reading2 

0.001*** 

0.001*** 

0.001*** 

0.001*** 

0.001*** 

0.001*** 


[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

lag_Reading3 

-0.000*** 

-0.000*** 

-0.000*** 

-0.000*** 

-0.000*** 

-0.000*** 


[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 

[0.000] 
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Table 2 (cont’d.) 


Stigma and Voucher Sanction Threat Impacts on i 

Reading Scores 

Variables 

(1) 

Grade 4 

(2) 

Grade 5 

(3) 

Grade 6 

(4) 

Grade 4 

(5) 

Grade 5 

(6) 

Grade 6 

x it 

FRL 

-17.10** 

-22.34*** 

-30.05*** 

-15.57* 

-24.78*** 

-32 19*** 

Disability 

[7.878] 

_67 99 *** 

[5.771] 

_54 97 *** 

[5.954] 

-61.48*** 

[9.189] 

-64.78*** 

[6.376] 

-46.10*** 

[5.761] 

-59.34*** 

Male 

[5.394] 

-22.17*** 

[6.471] 

-23.43*** 

[5.641] 

-15.90*** 

[6.269] 

-19 13*** 

[6.941] 

- 22 . 86 *** 

[5.461] 

-16.42*** 

Minority 

[3.761] 

-50.42*** 

[3.976] 

-20.74*** 

[3.123] 

-11.04** 

[4.291] 

-54.96*** 

[4.637] 

-22.97*** 

[3.331] 

-9.721* 

XT' 

[ 6 . 888 ] 

[6.056] 

[5.033] 

[8.069] 

[7.078] 

[5.267] 

Y t 

Y_2003 

52.78*** 

-4.378 

-82.64*** 

52.25*** 

-4.470 

-82.12*** 

Y_2004 

[11.33] 

56.08*** 

[11.24] 

8.515 

[11.74] 

-57 1 1 *** 

[11.31] 

55.37*** 

[11.32] 

8.564 

[11.76] 

-56.66*** 

Y_2006 

[17.58] 

-28.15* 

[14.30] 

57 . 34 *** 

[10.78] 

-42.20*** 

[17.60] 

-27.76* 

[14.26] 

57.76*** 

[10.75] 

-42.35*** 

Y_2007 

[15.09] 

12.12 

[12.17] 

26.35*** 

[11.55] 

-67.23*** 

[15.16] 

12.47 

[12.17] 

26.51*** 

[11.49] 

-67.43*** 

Y_2008 

[9.459] 

37 . 34 *** 

[8.485] 

44.33*** 

[9.852] 

-3.378 

[9.349] 

[8.488] 

[9.842] 


[9.461] 

[8.854] 

[13.10] 




Constant 

995.8*** 

1,209*** 

1,596*** 

1,003*** 

1,183*** 

1,607*** 


[31.99] 

[35.78] 

[37.12] 

[35.57] 

[38.93] 

[42.73] 

Observations 

13,265 

13,482 

19,833 

10,348 

10,587 

18,081 

R-squared 

0.526 

0.587 

0.564 

0.514 

0.577 

0.561 


Note: Robust standard errors in brackets. *** - significant at p < 0.01, ** -p < 0.05, * -p < 0.10 


Discussion 

We now discuss the results of our analyses and provide some policy implications. In the 
majority of our model specifications, elementary students attending schools that recently received 
grades of F make statistically significant gains in both math and reading. This finding corroborates 
with studies that have found that test-based accountability programs do indeed produce significant 
gains in students’ state assessment test-score gains (e.g., Carnoy & Loeb, 2002; Dee & Jacob, 2011; 
Figlio & Rouse, 2006; Hanushek & Raymond, 2005). When examining the interaction of a failing 
grade with the presence of a private school voucher sanction, we find no impact on math scores and 
a statistically significant decrease in reading. Therefore, our results also provide evidence to support 
the conclusions of Figlio and Rouse (2006), Ladd and Glennie (2001), and Harris (2001) that an 
accountability-based private school voucher sanction does not independently produce increases in 
achievement for students enrolled in lower-performing schools. 

There are at least a few possible explanations for these findings. First, the lack of effect for 
sanctions in math likely indicates that the possibility of losing students through private school 
vouchers does not incentivize schools in ways that necessarily increase test score results. However, 
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these schools are motivated to remove the reputational effects of a failing grade or benefit from the 
additional attention and resources that the state provides. It is also plausible that schools in this 
study did not take the private school voucher threat seriously due to low take-up rates in the years 
that the private school vouchers were available. The year with the highest Florida Opportunity 
Scholarship Program enrollment was 2004-05 with only 763 students statewide using the scholarship 
to enroll in private schools (Florida Department of Education, 2010b). This explanation is 
consistent with the findings from Goldhaber and Hannaway (2004) that school personnel were very 
much unaware of the private school voucher sanctions, especially relative to public scmtiny and state 
attention. Since teachers, principals, and district administrators did not really experience a massive 
student exodus with the offer of private school vouchers, it is unlikely that this sanction provided 
any significant, additional incentive. However, these explanations do not explain the negative effect 
of sanction threats on reading performance. 

Despite the exhaustive data available, we are not currently able to explain the negative effect 
of the threat on reading performance definitively and must rely on theory and intuition to guide this 
aspect of the discussion. One potential explanation for this finding is that school improvements 
implemented as a result of accountability initiatives take time before showing significant effects in 
student achievement. If student performance is improving over time as a result of these 
accountability measures, test scores would be highest in the most recent years. Therefore, it could be 
the case that the maturing and greater effectiveness, at least for the A+ Program as reflected in 
reading scores, happens to coincide with the removal of the private school voucher sanction, leading 
to a negative bias in the estimated impact of the private school voucher sanction. 

Another possible explanation is that something else, which is unobservable, changes when 
the voucher threat was removed, such as parental attitudes or involvement. Jacobsen, Saultz, and 
Snyder (2013) suggest that parental satisfaction and public support fall as schools go on the verge of 
sanctioning. Voucher threats potentially have a negative influence on school-community morale. For 
example, some parents might lose the motivation to be as active and work with these struggling 
schools to improve academic performance when vouchers are potentially available. Once the private 
voucher program is eliminated, school-community members might now have a stronger incentive to 
become more involved. The fact that this effect is found with reading and not math may support 
this interpretation since reading achievement is often more closely tied to parental involvement 
relative to other subject areas (Cronin, Kingsbury, McCall, & Bowe, 2005; Heyneman, 2005). 

Finally, there are also limitations to consider when examining the results of this study. While 
we wish to further inform the broader discussion on NCLB-type test-based accountability programs, 
it is important to note that our data are limited to Florida’s elementary-aged student test-score 
achievement over a six-year period. We also do not know the extent to which the private school 
voucher threat significantly impacts the performance of schools not near the cut point for being 
deemed as underperforming. It is likely that these schools on the bubble are the most likely to 
change significantly from the addition or removal of these types of sanctions, but we cannot say 
with certainty how private school voucher threats influence schools at different performance levels. 
Finally, we focus on achievement only in terms of standardized test score gains. Private school 
voucher sanctions could have significant impacts on other educational outcomes not examined in 
this particular study such as student retention, graduation rates, and college enrollment. 

Conclusion 

The findings from this study suggest that a state-operated private school voucher sanction 
does not incentivize lower-performing public schools in ways that increase students’ test scores 



Education Policy Analysis Archives Vol. 22 No. 87 


12 


beyond the effects from the stigma and attention that come with being deemed as underperforming. 
Despite the ineffectiveness of the private school voucher sanction, schools do appear to take action 
and improve on standardized test-based measures when school performances are graded and made 
public. 

There are some lingering issues that future research could address in order to still better 
inform the design and implementation of test-based accountability programs. While raising student 
achievement in terms of test score gains is a major concern for education stakeholders, researchers 
should evaluate the impact that the incorporation or discontinuation of private school voucher 
sanctions has on other important outcome measures, such as student grade promotion and degree 
attainment. The fact that there was a low private school voucher take-up rate in Florida also raises 
the question of whether this sanction would have greater impact with more students (and funds) 
actually leaving a public school district. Lastly, the finding of a negative effect in reading scores raises 
questions that beg for additional research on the mechanisms going on inside the black box. 
Unfortunately, we are not able to address these issues with our data, but hopefully future research 
will conduct analyses like that of the Goldhaber and Hannaway (2004) study to better address these 
important questions. 

The strong, positive response to a negative stigma (in addition to the null/negative impact of 
the voucher sanction) still has significant implications with regard to constructing optimal education 
policy. There is considerable debate about the role of private school vouchers in accountability 
programs. Some school choice proponents argue that these vouchers incentivize schools to perform 
better when faced with the possibility of losing students and funding, but many voucher skeptics 
contend that publicly reporting performance and the extra public attention and state resources are 
primarily responsible for the success of test-based accountability programs. Our findings indicate a 
state-funded private school voucher sanction does not significantly increase student achievement, 
independent of other aspects of an accountability program. While state-provided private school 
vouchers may not be a necessary component of effective school reform programs, there is, clearly, 
still much research to be done so policy makers can confidently design programs to use states’ 
education resources most efficiently. 
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Appendix A 


Table A1 


Descriptive Statistics for Fourth Grade Students in Schools Close to the D/F Cut Point 


Variable 

N 

Mean 

Std. Dev. 

Min 

Max 

DSS_Math 

13281 

1405 

249 

581 

2330 

Lag_Math 

13281 

1251 

283 

375 

2225 

DSS_Reading 

13235 

1412 

296 

295 

2527 

Lag_Reading 

13238 

1193 

318 

86 

2514 

FRL 

13281 

0.901 

0.299 

0 

1 

Disability 

13281 

0.210 

0.408 

0 

1 

Male 

13281 

0.505 

0.500 

0 

1 

Minority 

13281 

0.721 

0.449 

0 

1 

2003 

13281 

0.136 

0.342 

0 

1 

2004 

13281 

0.150 

0.357 

0 

1 

2006 

13281 

0.066 

0.248 

0 

1 

2007 

13281 

0.262 

0.440 

0 

1 

2008 

13281 

0.220 

0.414 

0 

1 

FLag 

13281 

0.248 

0.432 

0 

1 

F*Threat 

13281 

0.052 

0.222 

0 

1 


Note: DSS is the FCAT developmental scale score. FRL is a dichotomous variable for whether students have 
obtained free or reduced lunch status. Disability is a dichotomous variable for whether the student has been 
identified with a learning disability. For all regressions, the 2005-06 school year is the omitted year variable. 



Stigmas Without Sanctions 


17 


Appendix B 


Table B1 


Descriptive Statistics for Fifth Grade Students in Schools Close to the D/F Cut Point 


Variable 

N 

Mean 

Std. Dev. 

Min 

Max 

DSS_Math 

13493 

1507 

251 

569 

2456 

Lag_Math 

13493 

1352 

256 

569 

2330 

DSS_Reading 

13458 

1425 

312 

474 

2713 

Lag_Reading 

13441 

1363 

320 

295 

2638 

FRL 

13493 

0.899 

0.301 

0 

1 

Disability 

13493 

0.210 

0.407 

0 

1 

Male 

13493 

0.498 

0.500 

0 

1 

Minority 

13493 

0.732 

0.443 

0 

1 

2003 

13493 

0.178 

0.382 

0 

1 

2004 

13493 

0.125 

0.331 

0 

1 

2006 

13493 

0.062 

0.241 

0 

1 

2007 

13493 

0.260 

0.439 

0 

1 

2008 

13493 

0.215 

0.411 

0 

1 

FLag 

13493 

0.249 

0.433 

0 

1 

F*Threat 

13493 

0.063 

0.244 

0 

1 


Note: DSS is the FCAT developmental scale score. FRL is a dichotomous variable for whether students have 
obtained free or reduced lunch status. Disability is a dichotomous variable for whether the student has been 
identified with a learning disability. For all regressions, the 2005-06 school year is the omitted year variable. 
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Appendix C 


Table Cl 


Descriptive Statistics for Sixth Grade Students in Schools Close to the D/F Cut Point 


Variable 

N 

Mean 

Std. Dev. 

Min 

Max 

DSS_Math 

19796 

1499 

271 

770 

2492 

Lag_Math 

19796 

1489 

264 

375 

2291 

DSS_Reading 

19693 

1472 

311 

539 

2758 

Lag_Reading 

19707 

1395 

318 

295 

2713 

FRL 

19796 

0.886 

0.317 

0 

1 

Disability 

19796 

0.225 

0.418 

0 

1 

Male 

19796 

0.522 

0.500 

0 

1 

Minority 

19796 

0.711 

0.453 

0 

1 

2003 

19796 

0.132 

0.338 

0 

1 

2004 

19796 

0.260 

0.439 

0 

1 

2006 

19796 

0.011 

0.102 

0 

1 

2007 

19796 

0.402 

0.490 

0 

1 

2008 

19796 

0.089 

0.284 

0 

1 

FLag 

19796 

0.215 

0.411 

0 

1 

F*Threat 

19796 

0.103 

0.304 

0 

1 


Note: DSS is the FCAT developmental scale score. FRL is a dichotomous variable for whether students have 
obtained free or reduced lunch status. Disability is a dichotomous variable for whether the student has been 
identified with a learning disability. For all regressions, the 2005-06 school year is the omitted year variable. 
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Casey Cobb University of Connecticut 
Arnold Danzig Arizona State University 

Antonia Darder University of Illinois, Urbana- 
Champaign 

Linda Darling-Hammond Stanford University 

Chad d'Entremont Strategies for Children 
John Diamond Harvard University 
Tara Donahue Learning Point Associates 
Sherman Dorn University of South Florida 
Christopher Joseph Frey Bowling Green State 
University 

Melissa Lynn Freeman* Adams State College 
Amy Garrett Dikkers University of Minnesota 
Gene V Glass Arizona State LTniversity 
Ronald Glass University of California, Santa Cruz 
Harvey Goldstein Bristol University 
Jacob P. K. Gross Indiana University 

Eric M. Haas WestEd 

Kimberly Joy Howard* University of Southern 
California 

Aimee Howley Ohio University 
Craig Howley Ohio LTniversity 
Steve Klees University of Maryland 
Jaekyung Lee SUNY Buffalo 


Christopher Lubienski University of Illinois, Urbana- 
Champaign 

Sarah Lubienski University of Illinois, Urbana- 
Champaign 

Samuel R. Lucas University of California, Berkeley 
Maria Martinez-Coslo LTniversity of Texas, Arlington 
William Mathis University of Colorado, Boulder 
Tristan McCowan Institute of Education, London 
Heinrich Mintrop University of California, Berkeley 
Michele S. Moses University of Colorado, Boulder 
Julianne Moss University of Melbourne 
Sharon Nichols University of Texas, San Antonio 
Noga O'Connor University of Iowa 

Joao Paraskveva LTniversity of Massachusetts, 
Dartmouth 

Laurence Parker LTniversity of Illinois, Urbana- 
Champaign 

Susan L. Robertson Bristol University 

John Rogers University of California, Los Angeles 

A. G. Rud Purdue LTniversity 

Felicia C. Sanders The Pennsylvania State LTniversity 
Janelle Scott University of California, Berkeley 

Kimberly Scott Arizona State University 
Dorothy Shipps Baruch College/CUNY 
Maria Teresa Tatto Michigan State University 
Larisa Warhol University of Connecticut 
Cally Waite Social Science Research Council 

John Weathers LTniversity of Colorado, Colorado 
Springs 

Kevin Weiner University of Colorado, Boulder 
Ed Wiley University of Colorado, Boulder 

Terrence G. Wiley Arizona State LTniversity 
John Willinsky Stanford University 
Kyo Yamashiro University of California, Los Angeles 
* Members of the New Scholars Board 



Stigmas Without Sanctions 


21 


archivos analfticos de polfticas educativas 
consejo editorial 

Editor: Gustavo E. Fischman (Arizona State University) 

Editores. Asociados Alejandro Canales (UNAM) y Jesus Romero Morante (Universidad de Cantabria) 


Armando Alcantara Santuario Instituto de 

Investigaciones sobre la Universidad y la Education, 
UNAM Mexico 

Claudio Almonacid Universidad Metropolitana de 
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Pilar Arnaiz Sanchez Universidad de Murcia, Espana 

Xavier Besalu Costa Universitat de Girona, Espana 

Jose Joaquin Brunner Universidad Diego Portales, 
Chile 

Damian Canales Sanchez Instituto Nacional para la 
Evaluation de la Education, Mexico 

Maria Caridad Garcia Universidad Catolica del Norte, 
Chile 

Raimundo Cuesta Fernandez IES Fray Luis de Leon, 
Espana 

Marco Antonio Delgado Fuentes LTniversidad 
Iberoamericana, Mexico 

Ines Dussel FLACSO, Argentina 

Rafael Feito Alonso Universidad Complutense de 
Madrid, Espana 

Pedro Flores Crespo Universidad Iberoamericana, 
Mexico 

Veronica Garcia Martinez LTniversidad Juarez 
Autonoma de Tabasco, Mexico 

Francisco F. Garcia Perez LTniversidad de Sevilla, 
Espana 

Edna Luna Serrano Universidad Autonoma de Baja 
California, Mexico 

Alma Maldonado Departamento de Investigaciones 
Educativas, Centro de Investigation y de Estudios 
Avanzados, Mexico 

Alejandro Marquez Jimenez Instituto de 

Investigaciones sobre la Universidad y la Education, 
UNAM Mexico 

Jose Felipe Martinez Fernandez LTniversity of 
California Los Angeles, USA 


Fanni Munoz Pontificia Universidad Catolica de Peru 


Imanol Ordorika Instituto de Investigaciones 
Economicas — UNAM, Mexico 

Maria Cristina Parra Sandoval LTniversidad de Zulia, 
Venezuela 

Miguel A. Pereyra LTniversidad de Granada, Espana 

Monica Pini Universidad Nacional de San Martin, 
Argentina 

Paula Razquin UNESCO, Francia 

Ignacio Rivas Flores Universidad de Malaga, Espana 

Daniel Schugurensky Universidad de Toronto-Ontario 
Institute of Studies in Education, Canada 

Orlando Pulido Chaves LTniversidad Pedagogica 
Nacional, Colombia 

Jose Gregorio Rodriguez LTniversidad Nacional de 
Colombia 

Miriam Rodriguez Vargas Universidad Autonoma de 
Tamaulipas, Mexico 

Mario Rueda Beltran Instituto de Investigaciones sobre 
la LTniversidad y la Education, UNAM Mexico 

Jose Luis San Fabian Maroto LTniversidad de Oviedo, 
Espana 

Yengny Marisol Silva Laya Universidad 
Iberoamericana, Mexico 

Aida Terron Banuelos LTniversidad de Oviedo, Espana 

Jurjo Torres Santome LTniversidad de la Coruna, 

Espana 

Antoni Verger Planells LTniversity of Amsterdam, 
Holanda 

Mario Yapu Universidad Para la Investigation 
Estrategica, Bolivia 
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Dalila Andrade de Oliveira Universidade Federal de 
Minas Gerais, Brasil 

Paulo Carrano Universidade Federal Fluminense, Brasil 

Alicia Maria Catalano de Bonamino Pontificia 
Universidade Catolica-Rio, Brasil 

Fabiana de Amorim Marcello Universidade Luterana 
do Brasil, Canoas, Brasil 

Alexandre Fernandez Vaz Universidade Federal de 
Santa Catarina, Brasil 

Gaudencio Frigotto Universidade do Estado do Rio de 
Janeiro, Brasil 

Alfredo M Gomes Universidade Federal de 
Pernambuco, Brasil 

Petronilha Beatriz Gonsalves e Silva Universidade 
Federal de Sao Carlos, Brasil 

Nadja Herman Pontificia Universidade Catolica —Rio 
Grande do Sul, Brasil 

Jose Machado Pais lnstituto de Ciencias Sociais da 
Universidade de Lisboa, Portugal 

Wenceslao Machado de Oliveira Jr. Universidade 
Estadual de Campinas, Brasil 


Jefferson Mainardes Universidade Estadual de Ponta 
Grossa, Brasil 

Luciano Mendes de Faria Filho Universidade Federal 
de Minas Gerais, Brasil 

Lia Raquel Moreira Oliveira Universidade do Minho, 
Portugal 

Belmira Oliveira Bueno Universidade de Sao Paulo, 
Brasil 

Antonio Teodoro Universidade Lusofona, Portugal 

Pia L. Wong California State University Sacramento, 
U.S.A 

Sandra Regina Sales Universidade Federal Rural do Rio 
de Janeiro, Brasil 

Elba Siqueira Sa Barreto Fundacao Carlos Chagas. 
Brasil 

Manuela Terraseca Universidade do Porto, Portugal 

Robert Verhine Universidade Federal da Bahia, Brasil 

Antonio A. S. Zuin Universidade Federal de Sao Carlos, 
Brasil 




